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ABSTRACT 


Human systems integration (HSI) applies knowledge of human capabilities and 
limitations to design more efficient, effective and safe military systems. HSI can 
enable impressive lifecycle cost savings and performance gains when 
implemented. In practice, HSI activities are hampered by the complexity of 
human-technology integration issues. This thesis develops a simplified 
framework for understanding and assessing HSI throughout the acquisition 
process. 

An HSI Activity Model is presented to conceptualize HSI activity in military 
acquisition. Established human factors and human computer interaction theories 
are applied to develop a concise view of HSI in action. The core activity of HSI is 
summarized as the balancing of human capabilities and limitations with the 
affordances and constraints presented by system technology, to accomplish 
system objectives. 

A Comprehensive Human Integration Evaluation Framework (CHIEF) is 
then developed to provide the acquisition community with a viable tool for 
assessing HSI during acquisition. A measurement approach, rating scales and 
criteria, and a consolidated scoring matrix are developed based on lessons 
gathered from current system assessment measures, and refinement with HSI 
practitioners. 

If implemented, the HSI Activity Model and CHIEF offer the potential to 
increase HSI understanding and awareness, leading to improved system 
outcomes across the DHS acquisition enterprise. 
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EXECUTIVE SUMMARY 


Human systems integration (HSI) incorporates knowledge of human capabilities 
and limitations into the design of modern systems to make them more efficient, 
effective and safe (Booher, 2003). When implemented in a timely fashion, HSI 
can drive impressive cost savings and performance gains (Booher, 1997; Liu, 
Valerdi, & Rhodes 2009). In application, HSI is hampered by the complexity of 
human-technology integration issues (Newman, Bruseberg, Lowe, Borras, & 
Tatlock, 2008; Boff, 1990). This thesis presents two products to enhance 
understanding of HSI during military acquisition: a conceptual model, and an 
assessment framework. 

The HSI activity model conceptualizes HSI activity during military 
acquisition. Established human factors and human-computer interaction theories 
were applied to military acquisition to develop a concise view of HSI in action. 
The model summarizes the core activity of HSI as the balancing of human 
capabilities and limitations with the affordances and constraints presented by 
system technology to accomplish system objectives (Figure 1). 



Proposed HSI Metric 


TRL 

(Technology Readiness Level) 


Technology 


Humans 


SRL 

(Systems Readiness Level) 


Figure 1. Proposed HSI activity model. 
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The Comprehensive Human Integration Evaluation Framework (CHIEF) 
was developed to provide the acquisition community with a viable tool for 
assessing HSI during acquisition. Systems assessment measures, including 
NASA/DOD technology readiness levels (TRL), were analyzed to determine 
criteria for a successful measure. The measurement approach, rating scales and 
criteria, and a consolidated scoring matrix were developed in collaboration with 
faculty from the Naval Postgraduate School. Elements of CHIEF were then 
refined during a series of workshops with practitioners from the Coast Guard 
Human Systems Integration Division (CG-1B3). As a culminating exercise, the 
practitioners evaluated the usability of CHIEF by conducting a mock HSI 
evaluation for a current major acquisition program. The finalized CHIEF 
framework elements are presented in Figure 2. 
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Figure 2.Comprehensive Human Evaluation Framework (CHIEF) elements 
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Areas for expanded investment and research include the development of 
dedicated CHIEF software, identification of measures to support the framework, 
and refinement of rating scales and criteria based on organizational needs. If 
implemented, the HSI Activity Model and CHIEF offer the potential to increase 
HSI understanding and awareness during the acquisition process, leading to 
improved system outcomes across the DHS acquisition enterprise. 
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I. INTRODUCTION 


A. WHAT IS HUMAN SYSTEMS INTEGRATION? 

Human systems integration (HSI) is an amalgam of many fields, unified by 
a pragmatic purpose: to understand human capabilities and limitations and apply 
this knowledge to system design (Chapanis, 1996; Booher, 2003). HSI has been 
described as a management concept, a design philosophy, and a technical 
approach centered on human users of modern systems (Booher, 2003). The 
International Council on Systems Engineering (INCOSE) Handbook captures this 
purpose: “The primary objective of HSI is to ensure that human capabilities and 
limitations are treated as a critical system element, regardless of whether 
humans in the system operate as individuals, crews, teams, units, or 
organizations.” (Haskins, Forsberg, Krueger, Walden, & Hamelin, 2006, p. 326) 

HSI is deeply rooted in the field of human factors (HF), originating from 
studies of task efficiency among factory workers in the early 1900s. By mid¬ 
century, research in human factors engineering (HFE) enabled better 
understanding of how man interacts with technology (Figure 1). This knowledge 
was integrated into the increasingly complex technological systems that emerged 
from the Second World War (Chapanis, 1996; Tvaryanas, 2010). 


Figure 1. 



Human factors during WWII (from DVI Aviation, Inc) 
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Our expanding knowledge of human-technology interaction led to more 
diverse approaches in the latter half of the 20th century. New ways to integrate 
human considerations into systems development were conceived, and areas of 
research were expanded to involve multiple domains. Rather than focusing 
narrowly on individual workstation or task design, practitioners addressed staffing 
policies, training approaches, and the effects of health and safety on overall 
system performance (Chapanis, 1996, Tvaryanas, 2010). These considerations 
became a key factor in the evaluation of system performance. With the advent of 
the U.S. Army’s MANPRINT program in the late 1980s, HSI was incorporated 
into modern military system acquisition (Booher, 2003, Tvaryanas, 2010). 

Organizations across the DOD and federal government recognize different 
domains within the HSI discipline. This thesis adopts the seven domains 
recognized by the U.S. Coast Guard: human factors engineering, personnel, 
manpower, performance support and training, systems safety and occupational 
health, survivability, and habitability (Figure 2). 


Human Factors Engineering (HFE): Employed din ing systems engineering over 
the life of the program to provide for effective human-machine interfaces and to 
meet HSI requirements. 

Personnel: Define the human performance characteristics of the user population 
based on die system description and projected characteristics of target occupational 
specialties. Personnel attributes are design parameters 

Manpower - : The mix of military, civilian, and contract support necessaiy to 
operate, maintain, train and support the system. 

Performance Support and Training (PS&T): Develops options for individual, 
collective, and joint training for operators, maintainers and support personnel, 
consistent with FORCECOM policies and, where appropriate, base training 
decisions on gaining effectiv eness evaluations. Tire PM shall address die major 
elements of training, and place special emphasis on options that enhance user 
capabilities, maintain skill proficiencies, and reduce individual and collective 
training costs. 

System Safety and Occupational Health (SS/OH): This domain integrates 
across disciplines and into systems engineering to determine system design 
characteristics that can minimize the risks of acute or chronic illness, disability, or 
death or injury to operators and maintainers; and of equipment damage, failure or 

loss. 

Survivability: Addresses personnel survivability issues including protection 
against detection, fratricide. Chemical, Biological. Nuclear. Radiation and 
High-Yield Explosives (CBNRE) effects; the integrity of the crew compartment; 
and provisions for rapid egress. 

Habitability: Establishes requirements for the physical environment, personnel 
services (e.g., medical and messing), working and living conditions (e.g.. berthing 
and personal hygiene). 


Figure 2. USCG HSI domains (from Acquisition Directorate, 2010) 
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HSI is not a collection of discrete disciplines, but various domains sharing 
a set of concepts and similar language. In practice, the borders between domains 
are porous and their contents are interrelated (Booher, 2003). The labels 
differentiating domains are mainly a convenient taxonomy that suggests the 
perspectives from which human-centered design issues may be understood. 

B. HSI IN PRACTICE 

HSI contains a rich body of knowledge spanning more than a century of 
research in the social, physical, and cognitive sciences. HSI and HF research 
offers empirically-grounded understanding of human capabilities and limitations, 
and time-tested sociotechnical design principles to the acquisitions community of 
practice (Chapanis, 1996). HSI Practitioners employ this specialized knowledge 
with a robust suite of tools, techniques, and methods (Hale, Ching, Brett, & 
Rothblum, 2009) to investigate the integration of man and machine in modern 
systems. 

HSI practitioners work in integrated product teams (IPTs) across every 
phase of military acquisition (USCG Acquisition Directorate, 2010). They 
contribute early by providing input to mission need statements (MNSs) and 
operational requirements documents (ORDs). As a system’s functional and 
physical architecture begins to take shape, practitioners work across functional 
areas to incorporate human-centered design principles. In later stages, 
practitioners facilitate human-centered program management, contracting, and 
test and evaluation (T&E) activities. 

HSI practitioners come from diverse disciplines and professional 
backgrounds. Qualifications range from graduate and undergraduate degrees in 
the human sciences to professional certifications (Kleiner & Booher, 2003; 
Chapanis, 1996). While their backgrounds and expertise vary, HSI practitioners 
occupy a distinctive role in acquisition. They advocate for system operators, 
maintainers, and supporters by ensuring that systems attributes are designed to 
support their needs. 
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c. 


HSI REALIZATIONS 


Decades of HF and HSI have revealed a number of important realities 
concerning human-centered design. This section addresses several critical 
themes. 

1. It Is Hard to Get HSI Right 

Human beings are deceivingly complex, and the questions that arise in 
intermingling people with technology warrant very careful study. Anthropometry 
(the science of measuring the parameters of the human body) is but one 
example of human complexity. Properly derived anthropometric design ensures 
that a system will physically accommodate its intended users (Pheasant & 
Haslegrave, 2005). Anthropometric conformance is often addressed by applying 
the “5th—95th percentile” criterion (Blanchard & Fabrycky, 2011; NASA, 2010). 
This criterion mandates that a system accommodate all users that fall within the 
5th to 95th percentile (central 90 percent) for relevant physical measurements. In 
commercial application, the use of anthropometries is seen in the adjustability of 
driver seats to ensure that a wide range of customers can operate a vehicle’s 
controls effectively. A sampling of anthropometric measures common to 
commercial and military applications is shown in Figure 3. 



n 


411 

HI 


n 


416 


No. 

Dimension 

Min (cm. (in)) 

Max (cm. (in)) 

75$ 

Sitting height 

77 7(30 6) 

101 3(39 9) 

330 

Eye height, sitting 

66 5 (26 2) 

88 9 (35.0) 

529 

Knee height, sitting 

45.5(17 9) 

63.5(25.0) 

678 

Popliteal height 

33 0(13.0) 

50.0 (19 7) 

751 

Shoulder>elbow length 

29 6(11.6) 

41 9(16 5) 

194 

Buttock-knee length 

52 1 (20 5) 

69.9 (27.5) 


Figure 3. Basic anthropometric measures (from NASA, 2010) 
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Accounting for height, reach, leg length and other body measurements in 
design might appear to be a simple matter of data retrieval: find the applicable 
database (as in Figure 3), select the measurement (e.g., arm length), and design 
to accommodate the desired range of measures. In most cases a significantly 
more detailed anthropometric analysis must be applied. A person with long legs, 
for instance, will not necessarily have a long torso, or long arms, as Figure 4 
illustrates. 



Figure 4. Differences between measures for two individuals with 
identical seated height (from McCauley, 2014) 


Humans are not constructed with uniform proportions (Pheasant & 
Haslegrave, 2005). Humans are in fact so variable that the differences seen in 
Figure 4 are common across nearly every physical measure (Moroney & Smith, 
1972). Because human variability is so pervasive, simplistic approaches are 
often problematic. The development of 21st century military systems has 
confirmed that a refined, multivariate analysis is required to calibrate systems to 
users (Lockett, Kozycki, Gordon, Bellandi, 2005). Multivariate analysis (Figure 5) 
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accounts for all potential combinations of individual anthropometric measures, 
using specialized modeling software (Lockett et al., 2005; NASA, 2010). 



Figure 5. Reach envelopes calculated for different users 
(from Ergo-Link by Sun Group Design, LLC ) 

Consider that a task as simple as pressing a button on a console involves 
more than ten anthropometric measures (Pheasant & Haslegrave, 2005) and 
countless potential paths of movement. When the complexity of this physical task 
is coupled with the complexity of the cognitive processes that guide human 
movement, the challenging nature of designing human-technology interfaces 
becomes apparent. 

2. HSI Is Inevitable 

Jakob Nielsen, co-founder of the Nielsen-Norman consulting group and 
former vice president of research at Apple Computer, once had this to say 
regarding the testing of a new product: 

your system will be tested for usability even if you don't do so 

yourself. Your customers will do it for you, as they struggle to use 
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the system. Any usability problems found by users in the field will 
undermine your reputation for quality products and the resulting 
change requests will be about 100 times more expensive to 
implement than changes discovered by yourself in the early phases 
of the project. (Nielsen, 1993, pp. 7—8) 

The true customers in a military acquisition are military users. It is these 
operators, maintainers, and supporters who eventually transform an assembly of 
technologies we know as a “system” into a “capability” (Booher, 2003). Even in 
the case of unmanned or autonomous systems, living users contribute directly or 
indirectly to almost every function (Murphy and Shields, 2012). Like users of 
commercial products, military users determine a system’s success or failure. 
However, unlike civilian consumers, military users often have less discretion in 
selecting the equipment they use to complete their tasks. 

In the consumer world, a system poorly calibrated to a customer's 
capabilities and limitations may lead to poor product ratings and limp sales 
(Nielsen, 1993). But poor HSI in a military system may bring error, mishap, and 
injury that may otherwise have been avoided (Booher, 1997; Tvaryanas, 2006). 
Beyond the vicissitudes of corporate fortunes, mission outcomes, and thus 
human lives, hang in the balance. The nature of military HSI demands that we 
identify and remove system performance detractors to the full extent of system 
resources. 

Systems eventually end up in the hands of users. Poor design elements 
are exposed, and changes will inevitably result. The lesson for military systems is 
straightforward: changes will always be necessary to make products suitable for 
operational use; it becomes a simple matter of how much we want to pay. 

3. HSI’s Timing Is Critical 

As the Coast Guard and DHS develop increasingly complex systems 
under tighter fiscal and manpower constraints, timely and effective HSI 
involvement grows increasingly important (Boff, 1990; Chapanis, 1996; Booher, 
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2003). Solving problems in human-technology integration is easily deferred in 
the immediacy of ensuring that system technology is being developed as 
scheduled and within cost constraints. While meeting deadlines and schedule 
constraints may be hallmarks of a well-executed acquisition, early identification 
and implementation of HSI-driven design improvements are also critical to a 
program’s ultimate success (Booher, 2003, Miller et. al., 2003). 
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Figure 6. The cost of system changes over time 
(from the INCOSE handbook) 


As seen in Figure 6, postponing HSI has a price tag. The costs associated 
with delaying HSI-related design changes increase quickly as a system matures 
(Blanchard & Fabrycky, 2011; Haskins et al., 2006). For example, it may become 
apparent that the operating console on the bridge of a new Coast Guard cutter 
should be relocated to improve watch awareness and maintenance access. In 
the early stages of acquisition, the console can be relocated with mouse-clicks in 
the design software. If the change is delayed until production has begun, a formal 
engineering change must be initiated, requiring extensive management and 
contracting overhead, in addition to material and labor costs. Figure 7, adopted 
from Systems Engineering & Analysis, 5th edition, illustrates this point. 
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Figure 7. HSI cost effectiveness over time 
(after Blanchard & Fabrycky, 2011) 


HSI makes systems better by ensuring that technological systems marry 
well with the realities of their human users. Designing effective, efficient, and safe 
systems for our service members requires careful, comprehensive, and timely 
human-technology integration. As commercial industry teaches us, changes are 
inevitable as human and technological elements of a system come together. 
What is not inevitable, however, is incurring excess cost because of these 
changes. Deferring HSI to the late stages of acquisition is enormously expensive. 
In military acquisition, the choice is ours: how much will we ask taxpayers to pay? 

D. THE IMPETUS FOR THIS THESIS 

The Human Factors and Behavioral Sciences Division of the Department 
of Homeland Security (DHS) published “A Vision for Human Systems Integration 
in the U.S. Department of Homeland Security” in 2009. The article identified ten 
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keys to implementing HSI within DHS. First among these was the following: 
“Standardize and institutionalize the process for embedding HSI into the DHS 
Technology Readiness Level [TRL] framework” (Wilson, Malone, Lockett- 
Reynolds, & Wilson, 2009, p. 2).” 

DHS’s emphasis on integrating human concerns into technological 
development is not surprising. In the next chapter, our study of the NASA/DOD 
TRL scale will reveal a framework that enables technology to be managed across 
diverse acquisition disciplines in a timely and effective manner. By contrast, few 
frameworks exist for evaluating how users integrate with technology (Johnston et 
al., 2013; Phillips, 2010). Stated plainly, human-integration considerations are at 
a severe disadvantage in the acquisition process, as compared to technologically 
driven requirements. 

The USCG and DHS are not the only organizations faced with this 
challenge. Implementing HSI and HF effectively is a mounting concern within the 
Department of Defense (DOD) and other federal agencies, the U.K.’s ministry of 
defense (MoD), and the commercial sector (Johnston et al., 2013; Earthy et al., 
1999). Although developmental measures have been proposed, there is no 
widely accepted framework for evaluating HSI throughout the acquisition 
lifecycle. 

This thesis proposes a conceptual model of the core activity of HSI and a 
framework for assessing HSI during acquisition. If successfully implemented, 
these products are expected to illuminate human-technology integration issues, 
allowing improved outcomes across the USCG/DHS acquisitions enterprise. 

The next chapter examines the development and emergence of the 
NASA/DOD TRL scale, which is widely used for technology readiness 
assessment in federal acquisition. Recently proposed human-technology 
integration measures are also examined and compared with TRL to establish 
criteria for an HSI assessment framework. 
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II. A REVIEW OF CURRENT SYSTEM READINESS MEASURES 


While a number of measures have been developed to assess system 
readiness, assessing human-technology integration is a relatively new 
undertaking (Phillips, 2010; Johnston et al., 2013). In this section, the 
NASA/DOD TRL scale is analyzed and compared with two developmental 
human-technology assessment frameworks (SHARE and HRL). Important links 
between these assessment frameworks and the TRL scale are noted and the 
drawbacks of using the scale’s structure to assess HF and HSI are identified. 

A. LESSONS FROM THE EMERGENCE OF NASA/DOD TECHNOLOGY 

READINESS LEVEL SCALE 

The NASA/DOD TRL scale is now widely accepted in federal acquisition 
(Johnston et al., 2013, Mankins 2009). This scale focuses exclusively on 
technology, and makes no attempt to account for human contributions to a 
system (Phillips, 2010). Nevertheless, its conception and successful propagation 
throughout the acquisition community offers lessons for establishing a viable 
systems assessment measure. To understand the importance of the TRL scale, 
some historical context is required. 

During America’s quest to dominate space in the 1970s, NASA was 
charged with integrating cutting-edge technology into spacecraft at an 
extraordinary rate. At the time, technology readiness assessment (TRA) was the 
chief methodology for ensuring technological components were suitably mature 
for NASA missions (Mankins, 2009). Assessments were conducted at multiple 
points in the development lifecycle, requiring detailed technical requirements to 
be translated across government, commercial, and scientific circles. The need to 
develop a unified standard for technology readiness soon became evident, and 
what we now know as the TRL scale emerged (Mankins, 1995; Mankins, 2009). 

The current TRL scale divides technological readiness into nine simply 
defined levels. The scale produces “a discipline-independent, program figure of 
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merit” (Mankins, 2009, p. 1216) with which to assess and communicate the 
maturity of a new technology. This allows complex technology readiness 
requirements to be understood and applied across disciplines. In a recent 
retrospective, John Mankins, a 25-year veteran at NASA, highlighted the TRL 
scale as “the foundation of modern technology readiness assessment” (Mankins, 
2009, p. 1217). This foundation has shaped NASA technology development for 
more than three decades. The current version of the TRL scale appears in Figure 
8 . 
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Figure 8. The NASA TRL scale (from NASA, 2012) 


More than three decades after implementation, users of the TRL scale 
include DHS, USCG, DOD, and government organizations throughout the world 
(Mankins, 2009; USD (AT&L), 2011). 

The most convincing evidence of the TRL scale’s utility may be found in its 
absence. Consider the task of evaluating a new engine technology for an aircraft 
acquisition program without the benefit of such a scale. The new engine 
promises an appreciable performance advantage over older technology, but has 
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yet to be demonstrated in a similar application. Successful implementation of this 
technology would require program managers, systems engineers, operational 
testers, contract specialists, and experts in multiple engineering areas to evolve 
specific performance criteria for the new engine. Reaching agreement among all 
parties would require considerable time and effort. Today, we can expedite this 
process by identifying the desired TRL. The common benchmark set by the TRL 
scale can then be expanded upon as necessary within each discipline involved in 
the acquisition program. 

Current challenges in assessing HSI closely parallel those faced by NASA 
and other agencies before the TRL scale was devised. HSI activities must be 
coordinated across disciplines without the benefit of a standardized assessment 
framework (Phillips, 2010; Wilson et al., 2009). Instead, today’s HSI practitioners 
must define, prioritize, and translate the desired state of human-technology 
integration on an issue-by-issue basis. 

B. REVIEW OF HUMAN FACTORS-RELATED SYSTEM READINESS 

MEASURES 

A review of government, commercial, and academic literature reveals two 
relevant models for assessing HSI during system acquisition. Since both models 
are new to acquisition, limited information is available on their effectiveness in 
actual programs. In this section, the merits and limitations of these models are 
discussed and compared with the characteristics of the TRL scale. 

1. System for Human Factors Readiness Evaluation and Human 
Factors Readiness Levels 

In 2008, the FAA initiated a sweeping update of the U.S. air traffic control 
system with the NextGen program, promising a top-to-bottom overhaul of aging 
air traffic control technologies. The FAA quickly acknowledged the role of human 
factors in NextGen’s success and authored a Small Business Innovation Request 
(SBIR) seeking industry assistance to provide specific tools for this purpose. A 
two-phase contract was awarded to Design Interactive (Dl), Inc., for a software 
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tool to improve NextGen outcomes by standardizing and automating human 
factors assessment. Dl created two related products, System for Human Factors 
Readiness Evaluation (SHARE), a computer-based system for tracking identified 
HF issues, and Human Factors Readiness Levels (HFRL), a scale for prioritizing 
the resolution of these issues (Design Interactive, 2012; Johnston et al., 2013). 

The first iteration of SHARE defined nine levels of HF readiness, aligning 
HFRL directly with the TRL scale. This initial HFRL configuration defined key 
human factors activities, and then separated them into graduated levels, similar 
to the TRL scale. Dl’s initial concept for the scale is seen in Figure 9. 
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Figure 9. Initial concept for HFRL (from Design Interactive, 2012) 


At Phase I review, FAA officials identified several drawbacks to this 
approach. Critically, the process-focused levels did not address actual outcomes 
of HF analysis and testing. HFRL scores could increase based on the completion 
of HF analysis and testing alone, even if outcomes were not favorable. This 
meant that even if a high-priority HF deficiency was discovered in late-stage 
testing, the system’s HFRL rating would remain unchanged. The FAA concluded 
that the scale’s inability to capture the importance of HF issues, regardless of 
timing, could lead to inaccurate ratings (Design Interactive, 2012). 
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Based on FAA feedback, Dl consulted with government and commercial 
HF practitioners to improve SHARE and HFRL functionality (Design Interactive, 
2012; Johnston, et al., 2013). Dl quickly shifted SHARE development toward the 
assessment of actual human-technology performance, allowing for system 
evaluation irrespective of technology state or acquisition phase. Based on Dl’s 
findings, the second iteration of SHARE was implemented as an issue-based, 
risk-guided system for tracking HF readiness during system development. 

This second iteration enables users to create a profile for individual HF 
issues as they arise during acquisition. HF practitioners can then rate each issue 
using a standardized FAA risk-classification scale, and catalog the issue into a 
specific category (e.g., anthropometries, displays and controls). HFRL scores are 
tabulated across categories and across the system, based on highest (worst) risk 
scores (Design Interactive, 2012). The SHARE software displays these HFRL 
ratings in useful dashboard metrics, allowing FAA users to track HFRL across 
categories and over time. An early SHARE software concept screen is shown in 
Figure 10. 
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Figure 10. Early SHARE software concept (from Design Interactive, 

2012 ) 


SHARE offers a viable platform for assessing HF-reiated deficiencies, but 
has some limitations. Risk is well accepted in federal acquisition, but has 
inherent limitations for capturing the full spectrum of system performance (DOD, 
2006). In military acquisition, system development is managed to ensure that 
performance remains between threshold and objective values (Defense 
Acquisitions University, 2010). Risk focuses on issues that fall below the 
threshold values, and does not evaluate performance that is above this 
threshold. This is useful for prioritizing HF or HSI deficiencies (Harrison & 
Forster, 2003), but may be inadequate for a holistic assessment of HSI 
performance. 

Nevertheless, SHARE and HFRL offer a robust toolset for HF issue 
identification, classification, and resolution (Design Interactive, 2012: Johnston, 
et al., 2013). Dl’s shift from process-based metrics toward performance-based 
metrics during software development suggests a key lesson for the development 
of an HSI evaluation framework. SHARE appears well positioned to standardize 
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and improve HF evaluation across the FAA. For application to HSI assessment, 
the format and structure of SFIARE may be useful in FIFE and other domains, 
and warrants careful consideration. 

2. Fluman Readiness Levels 

The “Development and Initial Evaluation of Fluman Readiness Level (FIRL) 
Framework” (2010) was a pioneering effort to establish a meaningful FHSI metric 
in DOD acquisition. The FIRL concept was conceived by Flector Acosta of the 
USAF 711th Fluman Performance Wing, and developed into the FIRL framework 
by Major Eric Phillips, USAF. Phillips constructed FIRL as a progressive scale 
(similar to the TRL) to synchronize FHSI activity and outcomes in the DOD 
acquisition framework (Phillips, 2010). 

Phillips used the NATO human view architecture (NATO FIVA) as a 
conceptual basis for FIRL. Introduced by the UK MoD in 2010 to supplement 
existing acquisitions procedures, the NATO human view describes the primary 
areas of human consideration required for modern system design. (For a 
thorough discussion on the NATO human view, see the NATO Human View 
Handbook, 2008.) Using the NATO human view, Phillips outlined criteria for 
developing a human readiness level scale based on current acquisition practices. 
These criteria included 1) adherence to accepted risk management practices; 

2) implementation of FHSI as early as possible in the acquisition processs; 

3) emphasis on human-centered design that contributes to total system 
performance; and 4) capturing the incremental cost of FHSI over the course of 
acquisition (Phillips, 2010). 

To create FIRL levels, Phillips conducted an extensive mapping of FHSI 
activities, processes, and products. This mapping was then evaluated against the 
NATO human view and sequenced according to acquisition phase into nine 
levels (Phillips 2010). The resulting FIRL scale and descriptions are presented in 
Table 1. 
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Human Readiness Level 

Description 

1. Activation of Human Systems 

Integration: base-lining and commitment. 

Lowest level of socio-technical readiness. 
HSI infrastructure is setup within Systems 
Engineering planning. Total system 
analysis from both functional relationship 
and organizational perspectives occurs. 
Activity examples include front-end 
analyses, preliminary functional allocation, 
and initial HSI assessment and plan. 

2. Human Systems Integration analysis 
suite in support of component technology 
development. 

Significant HSI input to acquisition 
development and documentation occurs. 
Activity examples include initial human- 
machine interface assessment, generation 
of Target Audience Description, and 
threat/hazard assessment. 

3. Component human touch point 
resolution (i.e., human-system interaction, 
to include hardware and software): 
refining requirements thresholds. 

Multiple needs analyses and studies are 
conducted in support of requirement 
definitions. HSI domain assessments 
inform ongoing development actions, as 
well. Activity examples include human-in- 
the-loop analyses, sub-system hazard 
analysis, and HSI plan update and 
revision. 

4. Component human touch point 
engineering parameters and human 
performance indicators. 

Iterative evaluation and analysis of each 

HSI domain takes place and provides 
critical items of consideration bearing on 
system design. Activity examples include 
usability testing, development of human- 
centered source selection, and updating 
the human-machine interface assessment. 

5. Limited system human performance 
parameters/demonstration. 

Various HSI assessments and testing are 
performed to support the significant 
increase in fidelity of breadboard 
technology. This includes supporting “high 
fidelity” laboratory integration of 
components. Activity examples include 
examining safety and occupational health 
design features and cognitive task 
analyses. 

6. Field (relevant environment) validation 
of human performance prototypes. 

Representative model or prototype system 
is tested in a relevant environment. 
Evaluations of human performance 
embedded in demonstration system occur 
and HSI predictive models are updated. 
Activity examples include testing human 
reliability and usability of prototype in a 
high-fidelity laboratory environment or in 
simulated operational environment. 
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7. Final Developmental Test & 
Evaluation/human performance 
parameters. 

Significant HSI participation in system test 
events occurs. Iterative evaluation and 
analysis of each HSI domain continues as 
well. Activity examples include error and 
fault analysis to cover human error 
performance, equipment operability, safety 
procedures, and error recovery 
mechanisms. 

8. Operational Test & Evaluation/human 
performance parameters. 

Special human-centric analyses are 
conducted to update thresholds, 
objectives, and evolving criteria for 
Operational Test & Evaluation. Iterative 
evaluation and analysis of each HSI 
domain continues as well. Activity 
examples include system hazard analysis 
and HSI domain tradeoff 
studies. 

9. Sustainment: Initiation of capability gap 
feedback cycle. 

Extensive and iterative review and 
verification of fielded system begins, as 
well as post-product improvement 
evaluations for the next incremental builds. 
Activity examples include post-fielding 
training evaluation analysis and sustaining 
a hazard analysis for the fielded system. 


Table 1. HRL nine-level scale and definitions 
(from Phillips, 2010) 


HRL levels share essential features with the first HFRL concept proposed 
by Design Interactive. Both frameworks were constructed to coincide with the 
TRL scale and both focus on the completion of acquisition activities. HRL is 
hindered by many of the same drawbacks identified by the FAA in their 
evaluation of Dl’s initial work. Like HFRL, HRLs take a process-oriented 
approach. The HRL scale provides for standardization and increased awareness, 
but does not assess or account for HSI efficacy. This can lead to inaccurate 
ratings. Consider, for example, the detailed definition for HRL Level 5 (Table 1): 

Various HSI assessments and testing are performed to support the 
significant increase in fidelity of breadboard technology. This 
includes supporting high fidelity laboratory integration of 
components. Activity examples include examining safety and 
occupational health design features and cognitive task analyses. 
(Phillips 2010) 
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Even if all activities in this HRL level are completed, the extent to which 
human-technology integration has contributed to system performance remains 
unclear. Many questions remain unanswered, including: 

• Were the results of HSI assessments favorable? 

• Did the state of human-technology integration improve or degrade 
overall system performance? 

• What did testing, evaluation, and analysis reveal about the human 
contribution to system goals? 

These questions reveal an important difference between technology readiness 
measures (i.e., TRL) and potential human-technology integration measures. 
Technology outcomes are more compatible with the progressive trajectory 
provided by the TRL scale. Reliability analysis, for example, examines system 
performance over time based on individual technology components either being 
operable or not; mechanical components either perform or fail; software code 
segments function or crash, and tend to do so in predictable ways (Blanchard & 
Fabrycky 2011). When applied to humans, however, this approach can become 
problematic (Sanders & McCormick 1993). Subjective assessments are typically 
required to evaluate human performance, and human behavior can vary 
considerably over time. As will be discussed in the following chapters, assessing 
how humans interact with technology requires a change in the way we think 
about humans and technology components in complex systems. 

C. POTENTIAL SUCCESS FACTORS FOR THE HSI FRAMEWORK 

The TRL scale’s many attractive qualities have made it the “coin of the 
realm” for systems assessment (Mankins, 2009). As suggested by the limitations 
identified with HRL and SHARE, however, a different vehicle is required for 
assessing human-technology integration. The following attributes are suggested 
as criteria for a meaningful HSI assessment framework: 
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1. Simple and Discipline Independent 

The HSI framework must be easily understood. Simple, discipline- 
independent language will increase the framework’s acceptance, implementation, 
and application across disciplines. 

2. Surpassing Risk-Based Measures 

Measuring the contribution of HSI must go beyond risk-based approaches 
to encompass all effects on total system performance. Considering risk alone 
may not offer the breadth required for a complete assessment of HSI 
performance. 

3. Performance Focused 

The HSI framework must focus on evaluating the adequacy of the human- 
technology integration, measured in terms of total system performance. This 
criterion is a departure from previous approaches, which focus largely on process 
completion rather than performance outcomes. 

By incorporating the helpful qualities of the TRL scale and building upon 
the useful aspects of HRL and SHARE, a more comprehensive framework for 
human-technology integration can be envisioned. The following chapters explore 
these concepts and offer a framework for assessing human-technology 
integration. 
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III. CONCEPTUALIZING HSI ACTIVITY DURING SYSTEM 

ACQUISITION 


A. MODEL FOUNDATIONS 

This chapter develops a conceptual model for HSI in system acquisition. 
The model is based on three fundamental tenets: humans and technology are 
the primary actors in the system; accomplishing system objectives is the raison 
d’etre for the interaction between humans and technology; and, humans and 
technology have different patterns of emergence during system acquisition. 

1. Human and Technology Roles in Modern Military Systems 

The allocation of functions to either humans or machines is one of the 
most important design decisions in HF and HSI. As early as the 1950s, 
researchers suggested rules of thumb for assigning system tasks to the most 
capable party. The “Fitts list” (Figure 11), developed by the industrial 
psychologist Paul Fitts, is perhaps the best known HF design guideline (de 
Winter & DODou, 2011). 
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Figure 11. Man and machine capabilities circa 1950 
(from de Winter & DODou, 2011) 


As technological capability and the understanding of human cognitive 
processes have improved, the relative strengths and weaknesses of man and 
technology have been the subject of continual debate (de Winter & DODou, 
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2011; Dekker & Woods, 2002). In this thesis, the respective merits of humans 
and technology and their ideal confluence are set aside. What we gather from 
more than five decades of discourse is that the primary actors in the system are 
either people or technology (Blanchard & Fabrycky, 2011). Following this logic, 
functions that cannot be performed by one party must be carried out by the other 
as they work in tandem to accomplish system goals. 

2. Combining Humans and Technology to Accomplish System 

Goals 

In modern systems, the humans, technological tools, and system goals 
are inextricably linked (Kuuti, 1996; Shattuck & Miller, 2006). This relationship is 
the focus of activity theory (AT), originating from Soviet psychology in the 1970s. 
Activity theory asserts that to develop a meaningful understanding of human- 
technological activity, we must examine three fundamental aspects of interaction 
(Kuuti, 1996): 

• Subject : the user or user group, including operators, 
maintainers, and supporters 

• Tools : technological elements and artifacts used to accomplish 
system tasks 

• Object : the objective or goal of system activity 

By considering the interaction of all three elements, AT provides a basic 
framework for analysis (for a more nuanced discussion of AT, see Kuuti, 1996 or 
Kaptelinin & Nardi, 1996). 

AT is a foundational concept in user experience design and human- 

computer interaction, where it has informed the merging of humans and 

technology for more than two decades (Kuuti, 1996). We propose that AT offers 

similar utility within military acquisition. In system acquisition, requirements 

derived from broad system objectives guide system development, shaping the 

ways in which technology and users are integrated. This maps well to the 

theoretical elements proposed in AT: the “subject” is analogous to the system 

users, including operators maintainers, and supporters; “tools” are analogous to 
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the elements operated by users; and the “object” is the performance objectives of 
the total system. Figure 12 displays this relationship. 



Figure 12. Activity theory (AT) in military acquisition (after Kuuti, 1996) 


The transformation in Figure 12 depicts the interaction of subject and tool 
within the context of the object (the performance objective) to produce an 
outcome. In the acquisition of a military system, this transformation occurs as 
users interact with system technology to perform functions. This process 
captures the essence of HSI. Thus Figure 12 depicts the total system 
performance that results from human-technology interaction. 

B. A MODEL FOR HSI IN ACTION 

Bringing together the concepts above, a consolidated view of HSI in 
military systems acquisition is derived. HSI is represented as the merging of 
human and non-human elements, unified by system objectives. 

1. The Proposed HSI Activity Model 

The foundational concepts presented in this research can be unified into a 
conceptual model. As depicted in Figure 13, the human and technological 
elements of a system derive from separate origins and merge to accomplish 
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system objectives. The accomplishment of these objectives is measured by total 
system performance goals (at right). 



Figure 13. Proposed HSI Activity Model 


The system evolves throughout the SE process, as suggested by the 
movement from left to right in Figure 13. Technology matures over time as an 
acquisition proceeds. Humans “come as they are,” with preexisting capabilities 
and limitations. What technology permits (affordances) and what it does not allow 
(constraints), must align with both what a human can do (capabilities) and cannot 
do (limitations). Human and technological elements interact as the system takes 
shape, represented by the convergence of arrows in the model. Orange, green, 
and blue boundaries enclose the respective system assessment areas. 

2. Human and Technological Contributions to the System 

As defined by Norman, affordances are those fundamental characteristics 
that determine how a “thing” can possibly be used (Norman, 1988). We use the 

26 























term technology affordances to capture the ways in which technological elements 
can be used to perform system functions. For example, steering wheels afford 
turning, which creates mechanical leverage and controls movement. However, 
technologies also have needs relative to a system. The steering process requires 
human perception, strength, judgment, and other capabilities to perform its 
function. We define these external requirements as technology constraints. 
These affordances and constraints are revealed as a technology is introduced 
into system to supplement human capabilities and overcome limitations. 

Like technology, humans offer capabilities that can be used in system 
tasks such as physical strength, vision, audition, memory, and motor skills. 
Humans also have needs relative to a system, such as a breathable atmosphere, 
lighting, and protection from mechanical shock, vibration, and excessive 
gravitational forces (NASA, 2010). Man and machine alike have something they 
offer and something they need. Table 2 provides examples of this 
interdependency from the human side, at different levels of task complexity. 


Example Human Capabilities 

Example Human Needs 

Human 

Function 

Level 

Activity 
Theory Level 

Situation analysis; complex 
decision making; sense-making; 
option generation 

Decision support, contextualized 
sensor information; process control 

Job 

Operation 

Activity Level 

Situation awareness, decision 
making automation oversight, 
meta-system monitoring 

Control feedback; temporal 
indicators, system status, spatial 
orientation, mode awareness 

Duty 

Action Level 

Subsystem monitoring, rule 
interpretation, system calibration 

Sensory input (haptic, visual, 
auditory etc.); physical 
functionality, menu control 

Task 

Visual scanning, target association, 
parameter monitoring 

Physical access; physical 
accommodation; visual access 

Subtask 

Operation Level 


Table 2. Human capabilities and limitations 
(after Shattuck & Miller, 2006; Blanchard & Fabrycky, 2011; Nardi 

& Kutti, 1996) 
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3. Different Paths of Emergence during System Acquisition 

Human performance is essential to modern military systems (Booher, 
2003). Like measures of propulsion, mechanical strength, and computing power, 
measures of human physiological, sensory and cognitive characteristics are 
fundamental considerations for effective design (Miller et. al., 2003). The 
development of the FIM-92 “Stinger” man-portable air defense system offers a 
superb example. In the final stages of testing, the fully assembled system 
(sensors, targeting system, launcher, missile, etc.) was reported to have a kill 
probability of 60 percent. With a human operator included in the system, 
however, the actual probability of kill fell to 30 percent (Booher, 2003). Poor 
human-technology interfaces led to a significant decrement in performance. The 
lesson for system design is clear: humans and technology must both be 
considered as contributors to system performance (Booher, 2003). 

The FIM-92 example underscores a critical fact about humans and 
technology. Technology and technology capacity can be evolved relatively 
quickly over a period of months or years to meet system needs. Human 
capabilities, however, evolve at a much slower rate over millennia. 

Consider the human attributes relevant to a Coast Guard coxswain piloting 
a small boat. The coxswain’s muscular strength and anthropometric 
measurements are essential considerations for the tasks at hand such as 
steering, throttle adjustment, and control actuation. These physical attributes can 
be measured directly and objectively, and their underlying mechanisms are 
relatively well understood (NASA, 2010; Pheasant & Haslegrave, 2005). 
Alternatively, consider the attention capacity and situational awareness of the 
same coxswain. These attributes may be measured only indirectly (Miller, 
Crowson & Narkevicius, 2003; NASA 2010) and their underlying cognitive 
mechanisms are the subject of ongoing scientific study (Shattuck & Miller, 2006). 

By virtue of these diverse capabilities, HSI must encompass a wide variety 
of measures, each with computational limitations. HSI measures range from 
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nominal and ordinal, which have certain computational restrictions, to interval 
and ratio measures that allow added statistical precision (Stangor, 2010). 
Examples of different types of measures relevant to HSI are illustrated in Table 3. 


Scale 

Definition 

HSI/Acquisition Example 

Nominal 

A categorical variable; offers 
categorization but cannot be ranked 

Gender, ethnicity, operational specialty code, 
decision making variables, etc. 

Ordinal 

A qualitative variable with values that 
can be rank ordered; distance between 
values is not specified 

Operator usability rating, perceived level of 
difficulty, perceived operator fatigue, situation 
awareness score, color coding, etc. 

Interval 

Variable where differences in values 

reflect distance between the values; 
zero point is arbitrary 

Temperature, seating orientation (degrees), 
direction of airflow, operator intelligence 
quotient, etc. 

Ratio 

Variable with defined ratio values and 
a true zero point 

Pounds of force required, button relief, 
processing speed, luminance, etc. 


Table 3. Measurement theory for military HSI 
(after Stangor, 2010) 


Although the capabilities and limitations of individual persons vary widely, 
they do so within a relatively consistent range. Intelligence quotient, visual acuity, 
or grip strength, for example, vary widely among individuals yet fall within a 
predictable range across populations (Chapanis, 1996; Miller et. ai., 2003). 
Technology, by contrast, can be constructed from the ground up with attributes 
chosen to fulfill a specific purpose. The materials, types of technology, and 
underlying architecture of a system can be re-composed as needed (Blanchard & 
Fabrycky, 2011). Humans are far more constrained as a system variable. Human 
performance variability can be reduced to some extent through selection, 
training, and manipulation of factors affecting human states. Unlike technology, 
however, recomposing or re-engineering human capabilities is simply not a 
viable system design option. 

Humans and technology develop from different origins, and contribute to 
system functions in very different ways. For this reason, our model of HSI Activity 
incorporates separate pathways for Humans and Technology. Put simply, 
technology may or may not be ready for the systems we desire, but humans are 
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ready now. It is the responsibility of those who design the system to do so in a 
manner that provides the affordances and constraints that align with the 
capabilities and limitations of the humans who will operate, maintain and support 
the system. 

4. Advantages of the Proposed HSI Activity Model 

The proposed HSI activity model adds to existing HSI and HF models by 
portraying the core activity of HSI as a single figure (for more on founding HSI 
models, see Booher, 2003). This feature is essential for improving HSI 
understanding and awareness, as advocated in earlier chapters. The HSI activity 
model lays the groundwork for the HSI assessment framework proposed in 
Chapter IV. 

The HSI activity model doubles as a design paradigm. Adopting the view 
proposed in the model, technology affordances are considered in concert with 
their constraints, and human capabilities are considered in concert with the 
limitations they bring. This provides valuable perspective for design before the 
integration of user and technology is attempted. 
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1. Change in Projected 
Operators (E-6 to E-3) 


1. New Technology 
Introduced 


2. New Technology 
Functionality 


2. New Target 
Audience Description 


3. Revised User 
Capabilities & Limitations 


Figure 14. HSI Activity Model example application 


Figure 14 illustrates an application of the HSI activity model. Fluman 
capabilities and limitations are integrated progressively with technology 
affordances and constraints. First, human capabilities and limitations emerge, 
based on the selection of users (top left). As the human contribution emerges, 
technology is introduced, offering new functionality but also creating new 
demands on users (bottom center). The contributions and needs of both are 
integrated to achieve system performance goals (right). 

5. Limitations of the HSI Activity Model 

Military acquisition often includes the iterative introduction of new 
technologies over multiple years of system development. In addition, user groups 
(operators, maintainers, and supporters) may be recomposed to meet changing 
program needs. It is conceded that the updating of human and technological 
needs will probably not occur in the progressive, neatly discernable fashion 
depicted by the alternating gray arrows in Figure 14. 
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Like other HF models, the proposed HSI activity model is an abstraction of 
a complex and multifaceted process; the model has the virtue of simplifying and 
generalizing HSI activities so they can be readily understood (Norman, 1988; 
Parasuraman, Sheridan, & Wickens, 2000). The inherent tradeoff with this 
abstraction is that precise measures have limited applicability. This 
oversimplification is made with a deliberate purpose: increasing HSI 
understanding and awareness. The equally important task of measuring HSI will 
be explored in the next chapter. 
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IV. DEVELOPMENT OF A FRAMEWORK FOR ASSESSING HSI 
DURING SYSTEM ACQUISITION 


This chapter documents the creation of a framework for assessing HSI in 
system acquisition. The framework was created in consultation with NPS HSI 
faculty and refined with HSI practitioners from the Coast Guard Human Systems 
Integration Division (CG-1B3) during a series of workshops. As a culminating 
exercise, practitioners applied the framework to assess a current acquisition 
program, the U.S. Coast Guard Fast Response Cutter, during a mock design 
review. The resulting Comprehensive Human Integration Evaluation Framework 
(CHIEF) elements are presented at the conclusion of the chapter. 

A. STEP ONE: DEVELOPING AN HSI FIGURE OF MERIT 

A “figure of merit” (FOM) is used in engineering to develop a concise 
expression of the "degree of goodness" among competing alternatives during 
design (Lee, 2011). Developing a FOM allows engineers to establish a value 
hierarchy and evaluate design alternatives based on their relative merit in a 
particular design application. According to veteran employees at NASA, the TRL 
scale was constructed specifically as a FOM (Mankins, 2009). Early in this 
project, we decided to develop a full FOM for HSI. 

First, a value hierarchy was created for each HSI domain. The objective of 
this effort was to create a generalized, system-independent definition of HSI 
success and failure for each HSI domain. As a benchmark for success, HSI FOM 
language was developed to match the concise language employed by the 
NASA/DOD TRL scale. Statements describing the desired state and undesired 
state of each HSI domain were crafted. Concepts and language were 
synthesized from the Coast Guard Major Systems Acquisition Manual (MSAM), 
the Defense Acquisition Guidebook (DAG), the U.S. Air Force Human Systems 
Integration Handbook, the U.S. Army MANPRINT Handbook, and academic 
textbooks. 
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Developing TRL-like definitions for each HSI domain required a significant 
reduction in verbiage. This was achieved by removing overly specific phrases, 
context, and design instances from main definitions, and describing them 
separately as design considerations. Table 4 shows the resulting HSI FOM for 
the domains of personnel, manpower and PS&T. 


Domain 

Manpower 

Personnel 

Performance Support & 
Training 


The correct number and mix 

Human performance factors 

Competency requirements are 


of personnel required for 

are identified and matched to 

established to reflect end- 

<D 

system operation, 

system tasks and workload; 

user tasks and workload; 

+-> 

CO 

maintenance and support is 

criteria are developed to 

knowledge, skill and ability 

"O 

o> 

identified; manning reflects 

effectively recruit, select and 

gaps are appropriately 

’cn 

the full range of operations, 

train personnel for safe, 

identified and training system 

o 

work demands, operating 

efficient and effective system 

is developed to match end- 


conditions and deployment 
scheme for the system. 

operation. 

user learning needs. 


manpower estimates, 

Target audience descriptors, 

Knowledge, skills, aptitudes, 


individual competencies, team 

knowledge, skills, aptitudes, 

experiences, cognitive 

to 

competencies, occupational 

experiences, traits, cognitive 

abilities, cognitive style, 

C 

specialties, resilience, task 

abilities, cognitive style, 

personality factors; 

(J 

c 

capacity, net workload, watch 

personality factors; workload, 

competencies, task demands, 

o 

u 

schedules, operating 

cognitive demands, task 

cognitive demands, subject 

c 

.tap 

conditions, ship fill, 

reguirements, task duration, 

matter, audience, learning 

<D 

Q 

deployment schedules, 

taskfreguency, task design, 

styles, delivery mode, training 


chronic fatigue levels 

task environment, operating 
conditions 

infrastructure, instructional 
technology, instructional 
design, evaluation metrics 


The mix and/or number of 

Individual performance factors 

Competency requirements are 


personnel specified for the 

are poorly matched to system 

not matched with end-user 


system exceeds allowable 

tasks, workload and skill 

tasks and workload; 

<D 

manning levels; manning is 

requirements. Criteria are not 

knowledge and skills gaps are 

ru 

4-J 

co 

insufficient for the full range 

adequately established or 

inadequately understood or 

"D 

O) 

of operations, work demands, 

implemented for recruiting, 

defined; training system is 

i_ 

'to 

operating conditions and 

selecting, training and 

incompatible with end-users 

"O 

C 

deployment scheme of the 

retaining personnel that meet 

learning needs and 

3 

system. 

or exceed the required 
aptitudes and traits for the 
system. 

environment; delivery and 
retention of job-relevant 
knowledge, skills and abilities 
is inadequate. 

to 

O) 

DAG Chapter 6.3 

DAG Chapter 6.3 

DAG Chapter 6.3 

c 

O) 

AF HSIHandbook 

AF HSI Handbook 

AF HSI Handbook 

i_ 

<D 

M— 

MANPRINT Handbook 

MANPRINT Handbook 

MANPRINT Handbook 

CD 

CC 

CG MSAM 

CG MSAM 

CG MSAM 


Table 4. Figure of merit for three HSI domains 

34 
















The full HSI FOM including statements for all Coast Guard HSI domains, 
is included in Appendix A. As seen in the appendices, outcomes in each HSI 
domain were successfully categorized using the FOM approach. This process 
revealed that a key element of NASA TRL could be applied to assessing HSI: 
that is, succinct, generalizable desired states could be articulated in a way similar 
to TRL definitions. The simple, discipline-independent language desired for the 
HSI Framework was now available. 

1. Centering the Scale: Balancing Technology and Human Needs 
for Total System Performance 

Developing the HSI FOM required identification of desired and undesired 
states in each HSI domain. As a result of this process, the central theme 
presented in the HSI Activity Model was confirmed: optimal performance in each 
HSI domain was best described as the balancing of needs between humans and 
technology for the accomplishment of system goals. This was an important 
confirmation of the activity-based view of HSI presented in the last chapter. 

The manpower domain offered an interesting case study. Manpower 
activities in acquisition typically focus on crafting the correct mixture of personnel 
and positions to meet system performance needs. This typically entails matching 
the competencies and capacity of personnel to the work presented by the system 
(Defense Acquisitions University, 2010). Developing a FOM for this activity 
involved a critical question: For a particular system, what characteristics make a 
given manpower solution more or less desirable than another? 

In an era of diminishing budgets, cost typically anchors our assessment of 
acquisition alternatives. The preferred solution offers the desired system 
performance attributes at a given cost level. Thus, establishing a preferred 
manpower alternative requires that we establish the candidate that will lead to 
the best possible total system performance (Barber, 2011; Defense Acquisitions 
University, 2010). By examining the dynamic interplay between the capabilities 
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and limitations of humans and the affordances and constraints of technology, we 
can identify the most viable manpower alternatives. 

2. Assembling HSI Understanding: From Detailed Standards to 
Management Indicators 

Measuring HSI efficacy is a particularly challenging endeavor. The single 
domain of HFE alone requires diverse policies, standards and design criteria. 
The Department of Defense Design Criteria Standard-Human Engineering (MIL- 
STD-1472G) is evidence of this fact, with detailed standards governing every 
aspect of human-technology interaction across the full spectrum of military 
systems and environments. NASA’s Human Integration Design Handbook 
(NASA/SP-2010-3407) offers similarly comprehensive guidance, with more than 
1,000 pages of detailed standards. 

Despite their extraordinary level of detail and breadth, interpreting and 
applying these standards can be challenging (Chapanis, 1996; Lockett & Powers, 
2003). Consider the example of designing a single button for a control console. 
More than ten design parameters for the configuration of the button can be found 
in MIL-STD-1472G. Example parameters include button relief (depth of press), 
spacing, size, shape, resistance (force required for actuation), arrangement in 
relation to function, labeling, luminance (ambient light), lighting (button 
illumination), reflectivity and guarding. Figure 15 offers a sampling of these 
standards. 
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Dimensions <D. Diameter) 

Resistance 

Fingertip 

Thumb 

Palm 

Bare 

hand 

Gloved 
hand - 

Bare 

hand 

Gloved 
hand - 

Bare 

hand 

Gloved 
hand - 

Single 

finger 

Different 
fingers - 

Thu mb/p al 
m 

Minimum 

10 mm 
(0.4 m) 

19 mm 
(0.75 in) 

19 mm 
(0.75 in) 

25 mm 
(1.0 m) 

40 mm 
(1.6 in) 

50 mm 
(2.0 in) 

2.8 N 
(10 oz) 

1.4 N 
(5 oz) 

2.8 N 
(10 oz) 

Maximum 

25 mm 
(1.0 m) 


25 mm 
(1.0 in) 

- 

70 mm 
(2.8 in) 


11.0 N 
(40 oz) 

5.6 N 
(20 oz) 

23.0 N 
(80 oz) 



Displacement (A) 

Fingertip 

Thumb or palm 

Minimum 

2.0 mm (0.08 m) 

3-0 mm (0.12 in) 

Maximum 

6.0 mm (0.25 in) 

38 mm (1.5 m) 



Separation (S) 

Single finger 

Single finger sequential - 

Different finger - 

Thumb or palm - 

Bare 

Gloved 

Minimum 

13 mm (0.5 in) 

25 mm (1.0 in) 

6.0 mm (0.25 in) 

6.0 mm (0.25 in) 

25 mm (1.0 m) 

Preferred 

50 mm (2.0 in) 

- 

13 mm (0.5 in) 

13 mm (0.5 in) 

150 mm (6.0 in) 


Figure 15. Button configuration parameters (from MILSTD 1472-G) 


Detailed HSI and HFE standards offer essential guidelines for human- 
technology interface design. For their effective application, however, additional 
factors must be taken into account. In practice, system performance is shaped by 
user characteristics and the circumstances of the task (Chapanis, 1996; Kuuti, 
1996). To evaluate the efficacy of a human-technology interface design, we must 
extend our analysis to include other aspects of the task activity. 

Our discussion in Chapter III advocated an activity-based evaluation of 
HSI. To carry this out, we require specific measures: HSI tools, techniques, 
approaches, and methods (TTAMs) that capture the interaction of configured 
system technology with representative user characteristics, in a relevant work 
context. For brevity, we refer to these as aggregating measures. 
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A survey of HSI literature reveals a host of TTAMs suitable for use as 
aggregating measures (UK MoD, 2008; FAA, 2014). Dynamic anthropometric 
modeling (DAM) is an example within the HFE domain, as seen in Figure 6 of 
Chapter I. Simulated human users and configured technology are placed into a 
virtual prototype through computer modeling; the practitioner can then evaluate 
technology features, such as buttons or control configuration, in concert with 
relevant user characteristics, such as anthropometric measures or range of 
motion (Pheasant & Flaslegrave, 2005). Practitioners can then assess human- 
technology interaction outcomes for the system. 

Virtual or physical prototyping provides a convenient example of an 
aggregating measure in FIFE; however, suitable measures in other domains are 
also available. The fatigue avoidance scheduling tool (FAST), for instance, 
serves as an aggregating measure for the manpower domain. FAST uses the 
72-hour sleep history of individuals to predict their alertness in tasks that require 
vigilance (Eddy & Hursh, 2006). Manning levels can then be adjusted to ensure 
that the individuals performing the task are capable of the vigilance required. 

Once sufficient evidence of FHSI outcomes is established for the selected 
FHSI domain, the impact of human-technology interaction on total system 
performance can be assessed. Figure 16 illustrates an example of this 
measurement process for the HFE domain. HFE policies and standards form the 
basis for engineer and practitioner activity; system attributes are expected to be 
developed based on these standards. The users are evaluated as they interact 
with the system in the work context, using aggregating measures. These 
measures are then combined to formulate an assessment of the HFE domain in 
question, Evaluations of the other HSI domains are combined into a 
comprehensive HSI assessment, using the framework proposed later in this 
chapter. 
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Figure 16. The “Iceberg”: From HFE standards to HSI performance 


Our study of technology readiness assessment in Chapter II revealed a 
key lesson: system assessment measures must be suitable for a broad 
audience. We propose that this is equally true for assessing HSI. Acquisition 
program personnel often have limited exposure to the HSI discipline. Despite 
this, modern acquisition requires program managers, systems engineers, 
contracting specialists and others to acquire and apply some basic HSI 
knowledge. The proposed HSI evaluation framework aims to augment acquisition 
practitioner knowledge by offering a consolidated assessment of HSI that is 
rooted in quantitative data and empirically derived standards. 

B. STEP TWO: DEVELOPING A HUMAN INTEGRATION EVALUATION 
FRAMEWORK 

With the HSI FOM and measurement approach established, attention 

shifted to development of an assessment format. Assembling HSI information 

across multiple domains presented a challenge. A familiar HFE assessment tool 
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provided an analog to meet this challenge and inspired the initial format for the 
HSI Framework. 

1. Establishing a Format for HSI Assessment 

Assessment of HSI required a novel approach. Aggregating measures had 
to be consolidated across seven HSI domains and then presented in an 
integrated and understandable manner. A variety of HF and HSI frameworks, 
tools, and evaluation schemes were analyzed, including those discussed in 
Chapter II. The NASA task-load index (TLX) scale presented several interesting 
qualities pertinent to the task of assessing HSI. TLX was developed by NASA in 
the 1980s for assessing human workload in human-technology systems (Hart & 
Staveland, 1988). NASA’s current version of the scale is shown in Figure 17. 



Figure 17. NASA TLX scale (electronic version) 
(from Wikipedia commons, 2012) 
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The TLX scale assesses multiple task dimensions within a given context. 
Similarly, the goal of an HSI assessment is to capture the contributions of each 
HSI domain and assess their contribution relative to total system performance. 
The NASA TLX scale demonstrated that this type of multi-dimension evaluation 
could be applied successfully (Hart, 2006). This provided the inspiration for the 
initial HSI assessment framework. 

2. Development of an Initial HSI Assessment Framework 

As a first step toward constructing an evaluation framework, desired end 
states for each HSI domain were adopted from the HSI FOM. Definitions were 
crafted to focus the assessment of each HSI domain, similar to the questions 
provided for NASA TLX performance dimensions. Initial HSI domain definitions 
appear in Figure 20. 

The initial HSI framework borrowed heavily from the format and orientation 
of NASA TLX. Initial blackboard prototypes were evolved into a spreadsheet- 
based framework, as seen in Figure 18. HSI domain definitions were listed 
across the top of the matrix, with rating scales in the columns beneath each 
domain. A data field was provided beneath each domain definition for 
practitioners to list aggregating measures and results. 

Next, a qualitative, five-level rating scale was added to capture the 
contribution of each domain to overall system performance. This “total system 
performance implication” (TSPI) rates the extent to which the integration of 
human operators, maintainers, and supporters into the system enhances or 
degrades the total performance. The TSPI scale operationalized HSI 
performance into five categories (from worst to best): severe degradation, 
moderate degradation, borderline to mild degradation, enhancement, and 
optimizing, as seen in Figure 18. 
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Figure 18. Initial HSI framework (four domains) 


Criteria were developed to assist practitioners in assigning a total system 
performance implication rating based on the aggregating measures for each 
domain. To identify language capable of distinguishing performance within and 
across multiple HSI domains, an in-depth review of HSI standards and policies 
was conducted. Criteria were synthesized from a wide variety of standards 
including MILSTD 1472-G, NASA STD-3000, the Air Force HSI Handbook, and 
the U.S. Army MANPRINT Handbook. Potential evaluation categories were 
developed based on their ability to apply across HSI domains. The initial 
categories were identified as human-technology balance factor, human risk 
factor, and weight of contributing evidence. These categories and their rating 
criteria are shown in Table 5. 
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Balancing of Human & 
Technological Needs 

Human Risk Factor 

Weight of Contributing 
Evidence 

5 

Very Good. The limitations of 
human users are fully 
understood and accounted for 
across the system for the 
given domain; application of 
technology is tailored for 
optimal leveraging of human 
capabilities to fulfill 
technology constraints. 

Completely Acceptable. 

The system places humans 
users are well within 
acceptable limits for health, 
safety and welfare for the 
given domain; task design, 
work design, 

Validated as Positive. 

Domain performance meets 
or exceeds minimum 
thresholds, benchmarks and 
standards confirmed across 
the spectrum of end users, 
operating modes and 
conditions, in relevant 
environments. 

4 

Good. The limitations of 
human users are understood 
and accounted for in system 
design for the given domain; 
application of technology is 
tailored for effective use of 
human capabilities to fulfill 
technology constraints. 

Reasonably Acceptable. 

The system places humans 
within acceptable limits for 
health, safety and welfare; 
human functions, work 
design and work schedule 
contribute to effective and 
efficient and safe system 
performance. 

Verified as Positive. 

Domain performance meets 
or is predicted to meet 
minimum thresholds, 
benchmarks and standards, 
verified by prototype testing, 
high fidelity simulation, or 
similarly reliable means. 

3 

Borderline. The limitations of 
human users are not fully 
accounted for in the system 
design; the application of 
technology fails to leverage 
available human capabilities— 
or- borders on exceeding 
human capabilities. 

Borderline. The system 
places humans narrowly or 
partially within acceptable 
limits for health, safety and 
welfare, with occasional 
excursions from safe limits 
or safe work practices. 

Inconclusive. Domain 
performance may or may not 
meet applicable thresholds, 
benchmarks and standards; 
information derived from 
testing or simulation is not 
available -or- is inadequate 
to confirm compliance. 

2 

Poor. Human capabilities and 
limitations are poorly matched 
to system aggregated tasks 
and workload; technology 
unnecessarily impedes, 
impairs or poorly human 
capabilities with technology 
constraints. 

Unacceptable. The system 
places humans routinely 
placed outside of acceptable 
limits for health, safety and 
welfare for the given domain; 
major contribution to 
program risk. 

Verified as Negative. 

Domain performance 
thresholds, benchmarks and 
standards are not met -or- 
domain performance 
thresholds, benchmarks and 
standards are undefined, 
and therefore untested. 

1 

Very Poor. Human 
capabilities and limitations 
are very poorly accounted for 
in the system design; the 
application of technology 
unnecessarily and 
unacceptably impairs human 
capabilities to fulfill 
technology constraints. 

Extremely Unacceptable. 

The system routinely places 
humans outside of 
acceptable limits for health, 
safety and welfare for the 
given domain; high potential 
exists for mishap, injury or 
death. 

Validated as Negative. 

Domain performance has 
tested below threshold 
levels, as confirmed with 
actual system users in 
prototypes or in high fidelity 
simulation. 


Table 5. Initial HSI framework rating scale criteria 


The content and compatibility of scales across HSI domains were a 
subject of continuous discussion and refinement during the early HSI Framework 
development. This initial HSI Framework and rating scale established a starting 
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point. Further refinement of the framework required feedback from current HSI 
practitioners, as described in the next section. 

C. STEP THREE: REFINING THE FRAMEWORK WITH HSI 

PRACTITIONERS 

The initial CHIEF rating approach was refined during a series of four 
workshops with experienced HSI practitioners from the U.S. Coast Guard Human 
Systems Integration Division (CG-1B3). The panel consisted of six active-duty 
and civilian HSI personnel assigned to CG-1B3. Panel members represented all 
HSI domains and ranged in experience from 0-4 to 0-5 (military), and GS-12 to 
GS-14 (civilian). Practitioners participated remotely from CG-1B3 offices located 
at U.S. Coast Guard Headquarters in Washington, DC. Participation was strictly 
voluntary, and interaction was documented on a non-attribution basis. The 
workshop plan was reviewed by the NPS Institutional Research Board (IRB) and 
was determined not to be human subject research. However, the workshop 
participants were accorded the rights and privileges to which they would have 
been entitled had it been deemed human subjects research. 

The four workshops were hosted by NPS via virtual meeting software over 
a period of ten weeks. Participants were invited to join a virtual collaboration 
worksite where workshop presentation material, references, and participant 
uploads were made accessible. Sessions were jointly moderated by the author 
and NPS HSI faculty members. Workshops were scheduled according to CG- 
1B3 personnel availability and averaged approximately two hours in length. A 
screenshot of the workshop collaboration site and presentation materials for all 
workshops is contained in the appendices. 

1. Workshop One: Introduction and Framework Refinement 

The first workshop introduced participants to the research, processes, and 
products developed for this thesis. Six participants were in attendance and seven 
HSI domains were represented. Ground rules, objectives, and activities for the 
workshop series were presented for discussion. An overview of the HSI Activity 
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Model and the initial HSI Framework were presented, followed by a question and 
answer session. 

After the opening presentation, draft HSI domain definitions (Figure 18) 
were presented. Workshop participants were prompted with a scenario in which a 
senior acquisitions official had requested an abbreviated definition of their 
particular HSI domain, containing no jargon or buzzwords. Practitioners were 
then asked to refine the draft HSI domain definitions. 

A thirty-minute breakout session was conducted, during which the 
practitioners revised the initial HSI domain definitions. The revised definitions 
were submitted electronically and then presented to the group. The workshop 
concluded after consensus had been achieved and plans for a second workshop 
were discussed. The finalized HSI domain definitions appear in Figure 19. 


Manpower 

The mixture of required positions is balanced for the 
system's aggregated human workload, and users are within 
acceptable limits of endurance, fatigue, and skills 

Personnel 

Criteria are sufficiently established to identify, train and 
retain personnel capable of performing at or above the 
levels required by the system, within projected manpower 
limitations. 

Performance 
Support & 
Training 

The instructional system design develops and delivers the 
human performance needed for efficient, effective and safe 
system operation within manpower, personnel and system 
constraints 

Human 

Factors 

Engineering 

The system design optimizes the capabilities and 
limitations of human operators, maintained, and support 
personnel for efficient, effective, suitable, and safe system 
performance. 

Systems 
Safety & 
Occupational 
Health 

The system safeguards human users and equipment from 
acute and chronic hazards to health and safety to enable 
efficient, effective and safe system performance. 

Survivability 

The system safeguards human users from hazards, and 
provides efficient and effective means for emergency 
egress and escape. 

Habitability 

The system configuration and environment features 
support effective human performance by providing for the 
comfort, convenience, and quality of life of personnel. 


Figure 19. Finalized HSI domain definitions 
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2. Workshop Two: Refinement of Rating Scale Criteria 

Workshop Two focused on refining the rating scale criteria for the HSI 
Framework. Four of the Workshop One practitioners were in attendance, and six 
HSI domains were represented. The draft framework, proposed measurement 
approach (Figure 16), a proposed workflow, and the draft rating scale criteria 
were presented, followed by an extensive question and answer session. 
Practitioner input focused on the perceived strengths and weakness of the 
aforementioned materials based on their experience. 

Practitioners identified three central concerns stemming from the initial 
HSI Framework. First, the use of a five-level rating scale (Table 5) was 
considered excessive. A main point of contention was that the inclusion of two 
levels of performance above “acceptable” did not align favorably with current 
acquisition practices, which are based on objective and threshold values (for 
additional description of these values, see the Defense Acquisition Guidebook 
(DAG) 2011, Section 2.1.1). Second, the rating scale criteria (i.e., defined scale 
levels) were viewed by participants as containing too much detail, inviting overly 
subjective interpretation. And third, ratings across categories were evaluated as 
not sufficiently synchronized with the TSPI scale (Figure 18). In particular, 
practitioners felt that they could not clearly differentiate between acceptable and 
non-acceptable HSI performance, given the initial rating scale. 

Following the opening discussion, practitioners were provided draft rating 
scales and asked to revise the wording. Practitioners were informed they would 
be asked to rate a current system acquisition for an upcoming management 
briefing and to defend their HSI ratings based on the scale criteria. A thirty- 
minute breakout session was then held for discussion and analysis. The 
workshop adjourned with a request from the author for continued offline 
discussions to resolve the concerns raised. Recommended modifications were 
received from CG-1B3 within several days. Practitioners recommended the 
following adjustments: 
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• Rating scales should remain at five levels to avoid skewing the 
scale toward suboptimal HSI performance levels. 

• Use of the terms “threshold” and “objective” should be omitted in 
favor of more universal, acquisition-independent language. 

• Scale wording should be aligned such that the minimum level of 
acceptability occurs between levels two and three. 

• The “Weight of Contributing Evidence” category should be 
separated from direct rating scales and implemented separately. 

The HSI Framework was reformatted and refined based on adjustments 
requested by the practitioners. Refinements included altering the scale language 
to harmonize rating categories, removing intermediary scale criteria for simplicity, 
and isolating and renaming the “Weight of Contributing Evidence” category as 
the “Acquisition Performance Factor.” It was requested that this factor be added 
after the domain had been scored. An “Accommodation Factor” category was 
introduced to supplement the other scales based on its applicability across 
domains. The reformatted rating scales and criteria are presented in Table 6. The 
acquisition performance rating scale is presented in Table 7. 
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Balancing of Human & 
Technological Needs 

Human Risk Factor 

Weight of Contributing 
Evidence 

5 

Very Good. The limitations of 
human users are fully 
understood and accounted for 
across the system for the given 
domain; application of 
technology is tailored for optimal 
leveraging of human capabilities 
to fulfill technology constraints. 

Completely Acceptable. The 

system places humans users 
are well within acceptable 
limits for health, safety and 
welfare for the given domain; 
task design, work design, 

Validated as Positive. 

Domain performance meets or 
exceeds minimum thresholds, 
benchmarks and standards 
confirmed across the spectrum 
of end users, operating modes 
and conditions, in relevant 
environments. 

4 

Good. The limitations of human 
users are understood and 
accounted for in system design 
for the given domain; 
application of technology is 
tailored for effective use of 
human capabilities to fulfill 
technology constraints. 

Reasonably Acceptable. The 

system places humans within 
acceptable limits for health, 
safety and welfare; human 
functions, work design and 
work schedule contribute to 
effective and efficient and safe 
system performance. 

Verified as Positive. Domain 
performance meets or is 
predicted to meet minimum 
thresholds, benchmarks and 
standards, verified by 
prototype testing, high fidelity 
simulation, or similarly reliable 
means. 

3 

Borderline. The limitations of 
human users are not fully 
accounted for in the system 
design; the application of 
technology fails to leverage 
available human capabilities—or- 
borders on exceeding human 
capabilities. 

Borderline. The system places 
humans narrowly or partially 
within acceptable limits for 
health, safety and welfare, with 
occasional excursions from 
safe limits or safe work 
practices. 

Inconclusive. Domain 
performance may or may not 
meet applicable thresholds, 
benchmarks and standards; 
information derived from 
testing or simulation is not 
available -or- is inadequate to 
confirm compliance. 

2 

Poor. Human capabilities and 
limitations are poorly matched 
to system aggregated tasks and 
workload; technology 
unnecessarily impedes, impairs 
or poorly human capabilities 
with technology constraints. 

Unacceptable. The system 
places humans routinely 
outside of acceptable limits for 
health, safety and welfare for 
the given domain; major 
contribution to program risk. 

Verified as Negative. Domain 
performance thresholds, 
benchmarks and standards are 
not met-or-domain 
performance thresholds, 
benchmarks and standards are 
undefined, and therefore 
untested. 

: 

Very Poor. Human capabilities 
and limitations are very poorly 
accounted for in the system 
design; the application of 
technology unnecessarily and 
unacceptably impairs human 
capabilities to fulfill technology 
constraints. 

Extremely Unacceptable. The 

system routinely places 
humans outside of acceptable 
limits for health, safety and 
welfare for the given domain; 
high potential exists for 
mishap, injury or death. 

Validated as Negative. 

Domain performance has 
tested below threshold levels, 
as confirmed with actual 
system users in prototypes or 
in high fidelity simulation. 


Table 6. Finalized CHIEF rating scale criteria 


Acquisition Progression Factor 
(specific to each HSI domain) 

Behind 

The quantity or quality of 
contributing evidence leading 
to the domain score is less 
than expected/required for the 
current acquisition phase. 

Concurrent 

The quantity and quality of 
contributing evidence is 
commensurate with the 
acquisition phase. 

Ahead 

The quantity and quality of 
contributing evidence is more 
than expected/required for the 
current acquisition phase. 


Table 7. Acquisitions performance factor (APF) 
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3. Workshop Three: Framework Finalization 

Workshop Three was aimed at finalizing the rating-scale criteria and the 
design of the HSI Framework scoring matrix. Six of the Workshop One 
practitioners were in attendance, and five HSI domains were represented. 
Workshop participants were presented with the revised rating scale definitions, 
including the “Accommodation Factor” category. The reformatted rating criteria 
met with broad consensus. 

Candidate designs of the HSI Framework scoring matrix were then 
presented. Alternatives included two versions with vertical performance scales 
(similar to TLX), and one with a horizontal performance scale. Practitioners were 
asked to select the candidate they considered most effective at conveying HSI 
assessment results. The horizontal format (Figure 20) was selected by the 
practitioners. 



Figure 20. Selected scoring matrix with example ratings 
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The workshop concluded after the practitioners had nominated a test case 
that would be used in the final workshop to validate the HSI Framework. Past, 
current, and future acquisition programs were considered as candidates. A 
recently fielded system, the Coast Guard Fast Response Cutter (FRC) (Figure 
21) was selected for the case study. A poll revealed that all participants had 
recent work experience within the FRC acquisition. The recently completed 
Production Readiness Review (PRR) would serve as a scenario for the HSI 
Framework rating (for more information on the engineering review process, see 
the DHS Instruction/Guidebook 102-01-001, Appendix B - Systems Engineering 
Lifecycle). 



Figure 21. Coast Guard Fast Response Cutter (FRC) (from U.S. Coast 
Guard heartland.coastguard.DODIive.mil) 

4. Workshop Four: HSI Framework Test Case 

Workshop Four employed the HSI Framework to conduct a simulated HSI 
assessment of the FRC acquisition program prior to the PRR. Four out of the six 
original practitioners were in attendance, and six HSI domains were represented. 
The workshop focused on evaluating the usability of the HSI framework and 
establishing a preferred scoring method. 
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The workshop began with a briefing on the test scenario. Practitioners 
were asked to draw on their knowledge of specific HSI measures and test data at 
the time of the PPR to develop HSI Framework ratings and then present domain 
scores and justifications to the group. An example workflow for the HSI 
Framework was presented by the moderators. A thirty-minute breakout session 
was held to develop ratings for each HSI domain. CG-1B3 leaders compiled 
individual domain ratings for the FRC. Practitioners then presented justifications 
for each rating based on the HSI Framework rating scales and criteria. 

With HSI Framework domain scores established, the desired method for 
representing those scores became the focus of the discussion. The merits of 
various numerical approaches were presented by the domain practitioners and 
HSI managers. Several key areas were addressed, including the appropriate 
level of numerical accuracy (i.e., number of decimal places), desired technique 
for numerical averaging, and the possibility of weighting the domains. The 
workshop concluded with consensus on the following measurement approach: 

• Numerical scoring would be limited to whole numbers, consistent 
with the level of precision afforded by HSI measures. 

• The Acquisition Performance Factor (APF) would be assigned a 
plus or minus symbol (+/-), in order to indicate the trend of 
performance in each domain without affecting the numerical score. 

• Where possible, a numerical average should be avoided in favor of 
displaying a complete score across all domains. 

• When an average was desired (i.e., representation of all HSI 
domains scores was not possible), scores falling below the 
acceptable level should also be indicated. 

• The final framework should include the ability to declare selected 
domains to be not applicable for specific projects. 

D. FINAL VERSION OF THE CHIEF 

The final version of the HSI Framework assembles all the components 
refined during the four workshops: the scoring matrix, the finalized domain 
definitions, and the rating scales with guiding criteria. We refer to these 
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collectively as the Comprehensive Human Integration Evaluation Framework (or 
CHIEF) rating system. This rating system can be used to assess HSI efficacy 
across HSI domains in any system acquisition program (Figure 22). 


Comprehensive FLman 
Integration Evaluation Framework 
(CHEF) 


Project ID 


Need 

Analyze & i 

Obtain 

Produce & 


Select | 


Oploy 


MSI Daman 

Manpower 





Figure 22. Final CHIEF components 


The final version of the CHIEF rating system (at the top of Figure 22) 
includes other refinements from the final HSI Framework workshop. An indicator 
bar to display the current phase of acquisition (top) and information fields for the 
author and occasion for the rating (bottom left) were incorporated into the final 
design. Separate fields were included for the HSI domain rating and the 
acquisition performance factor. 
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The final version of the CHIEF rating system is configured as a “rolled-up” 
view, removing background information such as HSI domain descriptions and the 
aggregating measures seen in the initial HSI Framework. As determined in the 
final workshop, these fields remain hidden and can be is implemented as display- 
on-demand information when required. The rating scale criteria, aggregated 
measures, and APF scale are similarly hidden. 

As will be discussed in the final chapter, incorporating on-demand display 
of information was considered highly desirable for future versions of CHIEF. 

1. Suggested CHIEF Workflow 

Figure 23 shows the notional workflow for developing CHIEF ratings, as 
conceptualized by the author and NPS HSI faculty, and validated by practitioners 
during the final HSI Framework refinement workshop. 


| Ongoing Major Acquisition j 4 

/ 

initiates 

START _^_ 


„ informs-^ 6. Overall HSI Rating j 


1. Assessment of the Manpower HSI 
domain is required 


prompts 

\ 


2. Practitioner identifies 
relevant measures & gathers results 



- informs 


5. CHIEF Rating for Manpower 
selected with notes provided 


t 

input 

/ 






4. Results & remediation strategy 
(if needed) reviewed with program 


n 



u 



* 


shapes 



FAST Results 


Workweek Audit 


Legacy System 
crew survey results 


SME evaluation 




Figure 23. Example CHIEF workflow 


The notional workflow for the development of CHIEF ratings can be 
summarized in six steps: 


• Step One: The practitioner is assigned to evaluate HSI 

performance in the selected domain(s) for a given system. The 
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practitioner then reviews the consolidated domain definition to 
guide assessment. 

• Step Two: The practitioner applies his or her experience to identify 
aggregating measures appropriate for the system; data from these 
measures are then collected for analysis. 

• Step Three: The results from the data collected are consolidated 
and assessed using the CHIEF rating scales and criteria; the 
practitioner applies his or her knowledge to determine the impact of 
the selected domain(s) on total system performance (e.g., 
manpower in Figure 22). 

• Step Four: Once the CHIEF domain rating has been assessed, the 
results and a remediation strategy (as required) are reviewed by a 
competent authority. Issues are resolved (if possible) at the 
appropriate organizational level. 

• Step Five: The CHIEF domain rating is assigned, including a 
summary of aggregated measures, source documentation, and a 
resolution strategy for discrepancies (as required). 

• Step Six: The individual CHIEF HSI domain ratings are 
consolidated with other domains for a comprehensive CHIEF 
assessment. 

In addition to reducing the potential for error, a standardized CHIEF 
workflow offers other benefits. Administrative process controls can be 
implemented at any step, offering senior HSI practitioners and acquisitions 
managers insight into HSI activities as they evolve. A standard workflow also 
facilitates process automation through software, as discussed in the next 
chapter. 
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V. SUMMARY AND CONCLUSION 


A. SUMMARY 

This thesis documents the development of a simplified framework for 
understanding and assessing HSI across diverse disciplines of acquisition. 

The proposed HSI Activity Model was developed to conceptualize HSI 
activity in military acquisition. Established human factors and human computer 
interaction theory was studied and applied to develop a concise view of HSI in 
action during military system acquisition. In this model, we summarized the core 
activity of HSI as the balancing of human capabilities and limitations with the 
affordances and constraints presented by system technology to accomplish 
system objectives. 

Next, we developed the CHIEF rating system to provide a method for 
assessing HSI during acquisition. Existing system assessment measures, 
including TRL, were studied to determine criteria for a successful HSI 
measurement framework. A measurement approach, rating scales and criteria, 
and a consolidated scoring matrix were then developed for the framework in 
cooperation with HSI faculty from the Naval Postgraduate School. This initial 
version of CHIEF was refined during four workshops with Coast Guard HSI 
practitioners. As a culminating exercise, practitioners employed CHIEF to 
evaluate a major USCG acquisition program. The usability of the framework, 
notional workflow, and preferred scoring method were then refined. 

Our research produced a usable framework, but revealed a number of 
necessary assumptions, limitations, and practical considerations that must be 
taken into account. These are discussed in the following sections. 
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B. EXAMINING CHIEF ASSUMPTIONS AND LIMITATIONS 

The research, discussions, and scenarios carried out in CHIEF’S 
development required a number of assumptions and revealed some potential 
limitations. The following emerged as common themes: 

1. Subject Matter Expertise Is Essential 

Effective use of CHIEF depends on the expertise of HSI practitioners. As 
seen in the proposed workflow (Figure 23), a CHIEF evaluation requires in-depth 
knowledge, skills, and abilities in the HSI domain. This becomes most evident 
during selection of aggregating measures and during interpretation of 
performance results using CHIEF rating scales. Developing an overall CHIEF 
rating across domains requires knowledge of, and experience with, overarching 
HSI policies and procedures. Without the required practitioner expertise, CHIEF 
ratings may very well be inaccurate. 

2. Aggregating Measures Are Pivotal 

The accuracy of CHIEF is predicated on relevant, valid, and reliable HSI 
TTAMs. Measures must be available to practitioners in areas that require 
measurement; program resources must also be available for aggregating 
measures to be planned for and collected in a timely fashion during acquisition. 
Establishing reliable measures is essential to the successful implementation of 
CHIEF as the standard for assessing the extent to which human and 
technological components form a healthy, synergistic bond that will achieve total 
system objectives. 

3. Recognize Statistical and Representational Limitations 

The system-wide HSI assessment proposed in CHIEF will require a wide 
variety of measures. Alignment of composite aggregating measures across 
CHIEF ratings scales will also require subjective interpretation, as noted earlier. 
For these reasons, data derived from CHIEF should be regarded as ordinal data 
(Stangor, 2010). The numerical scores generated by CHIEF imply the ability to 
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calculate an overall HSI average for an acquisition program, but this temptation 
should be resisted. Since no weighting scale is applied during the CHIEF rating 
process, a straightforward averaging of scores suggests equal weighting of all 
domains, regardless of system realities. Using an un-weighted mean when all 
domains are not equal will result in a skewed perception of HSI efficacy in the 
acquisition program. 

CHIEF was conceived for conveying HSI understanding during key 
acquisition events. Thus, CHIEF results are expected to share limited space with 
other acquisition measures and information. Displaying individual CHIEF domain 
scores and supporting information may not be feasible. In circumstances where a 
composite HSI assessment is desired, using an average score based on the total 
system performance implication scale is suggested, supplemented with an 
indication of all domains rated below the acceptable level of performance (TSPI 
less than three). This will avoid underperforming domain scores from being 
obscured during averaging. 

C. RECOMMENDED DIRECTIONS FOR FUTURE RESEARCH 

The development of CHIEF was limited by the time and resources 
available for this thesis. Though functional, the elements of CHIEF require further 
refinement. Several promising areas for continued research are discussed below. 

1. Continue Developing the CHIEF Software 

Developing CHIEF to its full potential will require the framework to be 
placed into a usable format. The current spreadsheet-based version of CHIEF 
requires extensive user manipulation and has limited input and output. This was 
sufficient for research and rapid prototyping, but not for adoption as an accepted 
HSI business practice. Development of specific, dedicated software was beyond 
the scope of this effort. However, CHIEF’S development and refinement revealed 
many desirable attributes for future software application (see Appendix C). 
Overall, the workflow and design of Dl’s SHARE software tool offers an excellent 

starting point for developing a CHIEF software architecture. 
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2. Refine CHIEF Rating Scale Criteria 

One of the most challenging aspects of developing CHIEF was 
establishment of rating scale criteria. Criteria needed to be precise enough to 
discriminate between levels of HSI performance without extraneous detail. 
Criteria also needed to be compatible with aggregating measures across multiple 
HSI domains. The scales and criteria presented in this thesis were validated in 
the final workshop scenario, but with a relatively small group of practitioners in a 
controlled scenario. Additional subscales and criteria will likely be required to suit 
a broader range of practitioners, and more diverse evaluation scenarios. 

3. Identify Suitable Aggregating Measures 

As proposed in Chapter IV, aggregating measures evaluate as they 
interact with system components in relevant work context. The broad range of 
systems in modern military acquisition means that this context will vary widely. 
Enterprise software systems, major cutters, and unmanned aircraft, for instance, 
represent very different operational contexts and may require a different suite of 
HSI TTAM to capture desired measures. Using the CHIEF framework will require 
an organization to identify relevant measures for each specific type of system. 

D. CONCLUSION 

Carrying out effective HSI remains a critical challenge for the USCG, DHS, 
and DOD. This thesis breaks new ground to establish useful metrics towards this 
end. As the current body of research reveals, achieving the necessary integration 
of humans and technology is a complex and challenging task. The history of TRL 
suggests that decades may pass before human-centered system measures are 
embraced and fully incorporated into federal acquisitions. The best way to 
accelerate this timeline is to incorporate CHIEF into acquisition policy and require 
the use of CHIEF for each milestone decision. Similar to TRL ratings, if an 
acquisition program receives an unsatisfactory CHIEF rating, the program cannot 
proceed to the next phase until the integration of humans and technology is 
deemed acceptable. 
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We hope that this research and the products offered will expand 
awareness of how critical the human component is to modern systems 
acquisition. Our collective understanding of human-technology integration (or 
lack thereof) will determine the survival of our service members and shape our 
operational fortunes. 
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APPENDIX A - HSI FIGURE OF MERIT 


PART ONE 


Domain 

Manpower 

Personnel 

Performance Support & 
Training 


The correct number and mix of 

Human performance factors 

Competency requirements are 


personnel required for system 

including, cognitive, physical, 

established to reflect end-user 


operation, maintenance and 

and sensory capabilities, 

tasks and workload; 

CD 

support is identified; manning 

knowledge, skills, abilities, and 

knowledge, skill and ability 

CD 

-t— 1 

reflects the full range of 

experience levels are 

gaps are appropriately 

CO 

“O 

operations, work demands, 

identified and matched to 

identified; training system is 

CD 

i_ 

operating conditions and 

system tasks and workload; 

developed and match to end- 

CO 

CD 

deployment scheme for the 

criteria are developed to 

user learning needs. 

Q 

system, given current and 
projected manpower estimates. 

effectively recruit, select and 
train personnel for safe, 
efficient and effective system 
operation. 



Manpower estimates, individual 

Target audience descriptors, 

Knowledge, skills, aptitudes, 


competencies, team 

knowledge, skills, aptitudes, 

experiences, cognitive abilities, 

CO 

C 

competencies, occupational 

experiences, traits, cognitive 

cognitive style, personality 

i_ 

CD 

specialties, resilience, 

abilities, cognitive style, 

factors; competencies, task 

C 

manpower estimates, task 

personality factors; workload, 

demands, cognitive demands, 

u 

capacity, net workload, watch 

cognitive demands, task 

subject matter, audience, 

c 

CUD 

schedules, operating conditions, 

reguirements, task duration, 

learning styles, delivery mode, 

CO 

CD 

ship fill, deployment schedules, 

task frequency, task design, 

training infrastructure, 

Q 

chronic fatigue levels 

task environment, operating 
conditions 

instructional technology, 
instructional design, 
evaluation metrics 


The mix and/or number of 

Individual performance factors 

Competency requirements are 


personnel specified for the 

are poorly matched to system 

not matched with end-user 

CD 

-t— 1 

fO 

system exceeds allowable 

tasks, workload and skill 

tasks and workload; 

manpower for the system; 

requirements. Criteria are not 

knowledge and skills gaps are 

-t— 1 

CO 

manning is insufficient for the 

adequately established or 

inadequately understood or 

T3 

Ol 

full range of operations, work 

implemented for recruiting, 

defined; training system is 

l/l 

demands, operating conditions 

selecting, training and 

incompatible with end-users 

01 

T3 

and deployment scheme of the 

retaining personnel that meet 

learning needs and 

C 

=) 

system. 

or exceed the required 
aptitudes and traits for the 
system. 

environment; delivery and 
retention of job-relevant 
knowledge, skills and abilities 
is inadequate. 

CO 

CD 

DAG Chapter 6.3 

DAG Chapter 6.3 

DAG Chapter 6.3 

o 

c 

AF HSI Handbook 

AF HSI Handbook 

AF HSI Handbook 

CD 

!>_ 

MANPRINT Handbook 

MANPRINT Handbook 

MANPRINT Handbook 

cu 

4— 

CD 

CC 

CG MSAM 

CG MSAM 

CG MSAM 
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PART TWO 


Domain 

Human Factors Engineering 

Anthropometry 

Ergonomics 

Cognitive 

Engineering 

Macro Ergonomics 

Desired State 

Physical characteristics, 
capabilities and 
limitations of the 
expected user population 
are established; physical 
interfaces including 
seating, racks, 
workstations, worksites, 
accesses and touch points 
accommodate the full 
user population. 

Functional interfaces 
including controls, 
displays, workstations, 
worksites, accesses, 
labeling and touch 
points are configured 
for effective, efficient 
and safe operation 
given the expected 
user population. 

Cognitive interfaces, 
processes and 
information flows 
are optimized for 
end-user cognitive 
abilities; end-user 
cognitive resources 
including memory, 
attention and 
decision making 
anchor the system 
design. 

Cooperative 
interfaces are 
optimized to enable 
and enhance team 
performance, 
collaboration and 
communication; 
organizational 
interfaces are 

reflective of the 
projected operating 
conditions and 
deployment scheme 
of the system. 

Design Concerns 

User physical 
characteristics, 
proportions and 
dimensions, 

anthropometric surveys, 
physical strength, range of 
motion, range of 
perception; accessibility, 
adjustability, 

configurability, modularity 

Reaction time, 
attention, working 
memory; display 
placement, information 
management, 
eguipment 
arrangement, color 
coding, gestalt 
principles, labeling, 
grouping of controls, 
ease of use, usability, 
task load index 

Attention cues, 
cognitive workload, 
decision rules, 
decision support 
systems, situation 
awareness, mental 
models, cognitive 
skills and attitudes, 
warnings and 
alarms, function 
allocation (humans 
vs. automation) 

Job design, operating 
policies and 
procedures, watch 
floor and workstation 
arrangement; 

communication 
scheme, watch team, 
management 
interfaces, crew/team 
coordination, shared 
mental models. 

Undesired State 

Capabilities and 
limitations of the 
expected user population 
are not fully understood & 
applied; tasks and work 
areas are inadequately 
arranged, or are not 
configurable to 
accommodate the full 
user population. 

Physical and virtual 
interfaces are not 
configured for 
effective, efficient and 
safe operation; 
interfaces, controls or 
other touch points do 
not recognize and 
incorporate physical 
capabilities and 
limitations of expected 

users. 

Cognitive workload, 
processes and 
information flows 
are incompatible 
with, or exceed end- 
user cognitive 
capabilities; 
functional allocation 
(automation) poorly 
balances human 
cognitive abilities 
and system 
capabilities. 

Cooperative 
interfaces are 
optimized for team 
performance, 
collaboration, and 
inter-user 
communication; 
organizational 
interfaces reflect 
system operating 
conditions, 
constraints and 

limitations. 
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PART THREE 


Domain 

Systems Safety & 
Occupational Health 

Habitability 

Survivability 

Desired State 

The full extent of potential 
acute and chronic hazards to 
personnel are fully understood; 
hazards are reduced, 
eliminated or mitigated 
commensurate with expected 
operations to minimize risk of 
acute and chronic injuries to 
the expected user population. 

Living and working conditions 
are conducive to sustaining the 
morale, safety, health, and 
comfort of the user 
population; living and working 
conditions enhance personnel 
effectiveness and mission 
accomplishment, and 
encourage recruitment and 
retention. 

System is configured to reduce 
the risk of fratricide, detection, 
and the probability of being 
attacked commensurate with 
projected operations; users are 
able to withstand hostile 
environments without aborting 
the mission or acute chronic 
illness, disability, or death. 

Design Concerns 

Walking/working surfaces, 
work at heights, pressure 
extremes, lock-out/tag-out, fire 
hazards, and explosive risk; 
shock hazards, noise, light, 
vibration, chemical exposure, 
radiological exposure, 
repetitive motion; 
psychological risk factors 

Lighting, space, ventilation, 
and sanitation; noise and 
temperature control; ship 
motion; religious, medical, and 
food services guality and 
availability; berthing, bathing, 
and personal hygiene, non- 
operational connectivity 

Crew compartment integrity, 
egress capability, 
emergency/rescue access; 
armor, ballistic protection, fire 
protection, fire suppression, 
decontamination capability; 
fratricide, detectability, 
resilience, reparability 

Undesired State 

The system fails to reduce, 
eliminate or mitigate 
personnel hazards to an 
acceptable level for expected 
operations; system is not 
adequately configured to 
minimize risk of acute and 
chronic injuries to the 
expected user population. 

Living or working conditions 
inhibit the morale, safety, 
health and/or comfort of 
system end-users. Living or 
working conditions diminish 
personnel effectiveness and 
mission accomplishment, 
and/or diminish recruitment 
and retention. 

System fails to adequately 
reduce the risk of fratricide, 
detection, and the probability 
of being attacked in projected 
operations; users are unable to 
withstand hostile 
environments without aborting 
the mission and/or suffering 
acute chronic illness or injury. 

References 

DAG Chapter 6.3.4 

MILSTD 1472-G 

AF HSI Handbook 

MANPRINT Handbook 

CG MSAM 

DAG Chapter 6.3.4 

MILSTD 1472-G 

MANPRINT Handbook 

CG MSAM 

DAG Chapter 6.3.4 

MILSTD 1472-G 

AF HSI Handbook 

MANPRINT Handbook 

CG MSAM 


63 














THIS PAGE INTENTIONALLY LEFT BLANK 


64 
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WORKSHOP ONE PRESENTATION 


* * 

Development of a Comprehensive Human 

Integration Evaluation Framework (CHIEF) 



66 









67 













HSI in a Systems Acquisition Context 









Workshop Goals 

0 Refine and distill the framework definitions and 
measures (i.e. validity) 


o Refine the scale and rating criteria (i.e. 
reliability ) 

0 Determine the potential utility of the framework 
for acquisition (i.e. usability) 


Desired End-state : CAE, MDAs & PMs incorporate 
HSI framework into formal policy 
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@r NASA/DOD Technology Readiness Level 


Actual syslom ‘flight proven" through successful 
mission operations 

Actual system completed and “flight qualified' 
through test and demonstration (Ground or Flight) 
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Rating Scale- Expanded View 



Questions : 

How does the system 
treat humans? 


How well does the 
system balance 
technological versus 
human needs? 


How certain are we? 
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C.H.I.E.F. Assumptions 

o Based on current state of the system 
at any point in acquisition process 



o Assumes technology will function as 




75 





































Workshop Exercise 1: 


You are approached by a senior acquisition executive (CAE 
or similar), who wants to get smarter on HSI. They ask you: 

‘ Could you send me a 1-2 sentence summary of the 

_[domain] of HSI work...summarized in your own 

words, without buzzwords or jargon? Atthe center of it all, 
what are you trying to accomplish in_[domain]? 


mk 


Your excellent staff has provided you with a draft. Send 
your edits via email. 


76 












77 








WORKSHOP TWO PRESENTATION 



Development of a Comprehensive Human 
Integration Evaluation Framework (CHIEF) 


Workshop 2: Rating Scale Criteria 
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Workshop 2 Goals 

0 Refine The Scale And Rating Criteria 

□ Rating Scale : Do we have the right number of 
categories? 

□ Scale Considerations : Are we answering the key 
questions? 

□ Criteria : Do we have the right scale criteria? 

Desired End-state : Give CG-1B3 practitioners a credible 
and reliable tool to advance human design priorities. 






Example of Measuring HSI: 

Nesting of standards during design of a control console 
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WORKSHOP THREE PRESENTATION 
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•/ 


problem 

statement 

The acquisitions community would benefit from 
a common framework to understand, assess 
and communicate HSI. 
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• Minimal acceptability at rating of 3 

• Middle definitions removed 

• Ratings earned by fulfillment of all criteria (similar to OER/EER) 


Knowledge Factor for Weighting - Glide Slope' Approach 


Acquisition Progression Factor (Domain Spaciftc) 
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JCC*ad to tha Ooma^i score lassas^ on *adom~ score 


• Centered on line of minimal acceptability as zero 

• Example : 

> HFE Score of 2 (not good) with AP Factor Behind (-) 
nets a domain score of 1.8 points out of 5 
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Look & Feel C 
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WORKSHOP FOUR PRESENTATION 



Development of a Comprehensive Human 
Integration Evaluation Framework (CHIEF) 
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Example - Measuring HFE 
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Semi-final Rating Scales & Criteria 



• Minimal acceptability at rating of 3 

• Ratings assigned by fulfillment of all criteria in each block 
(similar to OER/EER) 


Acquisition Progression Factor - Glide Slope' Approach 


Acquisition Progression Fee tor (Domain Specific) 

(Oo wo havo tho syskKTvspoc/< HSI knowledge that wo would sNxrfd lor the gvon ooqutvbon phtse?) 

Behind {•) 

The quantity <x quality of contnbuong 
evidance leadng to Be Sornim tcora it 
lett Mn inpectedrnyjred lor the 
current acquueon phate 

Concurrent (•) 

The quantity andquafcty of 
corrrc^tng endanca it 
commemorate mu the acqwvocn 
phate 

Ahead (♦) 

The quarety and quatty of ccntnbutng 
evidence it more than 
eipected. required lor the current 
ecqumtion phate 


• In practice : 

> We don’t know as much as we should = AP Factor Behind (-) 

> We are on track’ for HSI knowledge = AP Factor neutral 

> We know more than we would expect to at the given acqu . 
phase = AP Factor Behind (+) 
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Workflow Example 
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1 


Data Collection : 

1) What rating and Rating and APF did you 
arrive at for your domain? 

2) How did you arrive at this rating? 

3) What measurements or analysis did you 
rely on? 

4) How easy or difficult was it to match your 
measures in the CHIEF rating scale? 
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APPENDIX C - DESIRED CHARACTERISTICS FOR CHIEF 

SOFTWARE 


• Basic Functionality : CHIEF software will provide a web-based 
graphical user interface and database for collecting and assessing 
HSI information during system acquisition. Users will be able to 
create, edit, approve and track HSI performance ratings and 
supporting information for federal acquisition programs, based on 
the CHIEF. 

• Expected User Group : The CHIEF user group will include HSI 
practitioners, HSI managers and other acquisition professionals 
from a variety of disciplines. Users will include federal government 
employees, military members, military contractors and vendors. 

• Interface and Display Modes : Information entered into CHIEF will 
serve different managerial levels, across multiple functional groups. 
CHIEF software will include specific interface and display modes to 
suit the needs of these user sub-groups. CHIEF information (e.g., 
HSI domain definitions, rating scale criteria, rating supporting 
information) will be rapidly accessible as “display-on-demand” 
information using hyperlinks, mouse-over or similar software 
functionality. 

• Information Input : HSI practitioners will carry out the bulk of CHIEF 
data entry, edits, and updates. File formats are expected to include 
text files, spreadsheets, photographs, video, scans, and electronic 
presentations. Additional formats may be required. 

• Information Validation : HSI managers will validate individual HSI 
domain ratings and supporting information entered by HSI 
practitioners. Acquisition managers will access, review and approve 
top level information. 

• Information Export: Reports and summary information from CHIEF 
software must be exportable in common formats including 
commercially-available word processing, spreadsheet, presentation 
and document display software. 

• Workflow Automation : Data entry will be implemented according to 
the standardized CHIEF workflow, using government approved 
reference materials and standards. The software should automate 
the workflow through the use of pre-positioned information, drop¬ 
down menus, links or similar functionality. 
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• Access Control : An access hierarchy will be required to manage 
each user’s ability to view, edit and control CHIEF information. 
Access to CHIEF software will be controlled by the appropriate 
authority within the government acquisitions organization. 

• IT Enterprise Compliance: CHIEF software will reside on 
government equipment and networks. Users and administrators will 
be expected to have access to the Coast Guard data network or 
analogous DOD systems (as appropriate). 
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